Skip to content

fix: 2 corrections on gbfs populate#1265

Merged
jcpitre merged 9 commits into
mainfrom
1242-gbfs-feeds-are-not-being-removed-from-api
Jul 9, 2025
Merged

fix: 2 corrections on gbfs populate#1265
jcpitre merged 9 commits into
mainfrom
1242-gbfs-feeds-are-not-being-removed-from-api

Conversation

@jcpitre

@jcpitre jcpitre commented Jun 17, 2025

Copy link
Copy Markdown
Collaborator

Closes #1242
Closes #1268

The two corrections are:

  • Deleted feeds from the DB if removed form systems.csv
    • There was code that would filter out feeds that were in in the DB but not in systems.csv. Thus we did not know that some feeds were removed from systems.csv
  • Stopped processing feeds based on changes in location
    • The location is initially set with the populate scripts from systems.csv, but then it is changed by the function that determines the location from the data. So comparing locations to see if something changed in CSV did not work.

For deletion, this PR also setup cascade deletes on certain DB tables in the SQL schema. For example, id a GbfsFeed is deleted, the entry in gbfsversion table referring to this feed is also deleted. This prevent failing of the deletion because gbfsversion.feed_id cannot be null and is a foreign key to gbfsfeed.

Testing tips:
See test_populate_gbfs.py for the tests that have been done.
We miodified the way the DB is filled in the api integration tests to allow 2 .csv files to be loaded. This allowed to test loading a csv in a DB that is already populated, as was necessary for this PR.

Please make sure these boxes are checked before submitting your pull request - thanks!

  • Run the unit tests with ./scripts/api-tests.sh to make sure you didn't break anything
  • Add or update any needed documentation to the repo
  • Format the title like "feat: [new feature short description]". Title must follow the Conventional Commit Specification(https://www.conventionalcommits.org/en/v1.0.0/).
  • Linked all relevant issues
  • Include screenshot(s) showing how this pull request works and fixes the issue(s)

@jcpitre jcpitre linked an issue Jun 17, 2025 that may be closed by this pull request
@jcpitre jcpitre marked this pull request as draft June 17, 2025 15:26
@jcpitre jcpitre marked this pull request as ready for review June 19, 2025 15:26
@jcpitre jcpitre requested a review from cka-y July 7, 2025 18:54

@cka-y cka-y left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM


# Exclude 'Location' from comparison because the DB values might have been changed in the
# python function that calculates the location.
columns_to_compare = [col for col in df_db_common.columns if col != "Location"]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

Comment thread api/src/shared/database/database.py Outdated
@jcpitre jcpitre merged commit f00a829 into main Jul 9, 2025
3 of 6 checks passed
@jcpitre jcpitre deleted the 1242-gbfs-feeds-are-not-being-removed-from-api branch July 9, 2025 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Modify DB schema to handle entity deletion properly GBFS feeds are not being removed from API

2 participants